Are you an aspiring Big Data Engineer or Developer interested in creating Data Pipelines for serving Data Warehouses and Data Analytics platforms? Would you like to learn all about ETL and ELT data pipelines and how to build them using Bash scripting and cutting-edge open-source tools such as Apache Airflow and Apache Kafka? This course may be just right for you.

ETL stands for Extract, Transform, and Load. It refers to the process of curating data from multiple sources and preparing the data for integration and loading into a destination platform such as a data warehouse or analytics environment. ELT is similar but loads the data in its raw format, reserving the transformations for people to apply themselves in a ‘self-serve analytics’ destination environment. Both methods are typical examples of data pipeline deployments.

In this course, you will explore the fundamental principles and techniques behind ETL and ELT processes. You will learn how to construct a basic ETL data pipeline from scratch using Bash shell-scripting. You will also learn about the tools, technologies, and use cases for the two main paradigms within data pipeline engineering: batch and streaming data pipelines.  You will further cement this knowledge by exploring and applying two popular open-source data pipeline tools: Apache Airflow and Apache Kafka.

You will learn all about Apache Airflow and use it to build, put into production, and monitor a basic batch ETL workflow. You will implement this data pipeline using Airflow’s central construct of a DAG (directed acyclic graph), consisting of simple Bash tasks and their dependencies.

You will also learn about Apache Kafka and use it to get hands-on experience with streaming data pipelines, implementing Kafka’s message producers and consumers, and creating a Kafka weather topic.

This video and hands-on, lab-based course is made up of 5 modules: Data Processing Techniques; ETL & Data Pipelines: Tools and Techniques; Building Data Pipelines using Apache Airflow; Building Streaming Pipelines using Apache Kafka; and a Final Assignment. “ETL and Pipelines” will provide you with a great hands-on introduction to the latest technologies in data pipeline engineering.